DOCUMENT RESUME 



ED 342 237 



FL 020 088 



AUTHOR 
TITLE 



PUB DATE 
NOTE 



PUB TYPE 



Paribakht, T. Sima; Wesche, Marjorie Bingham 

A Methocaology for Studying the Relationship between 

comprehension and Second Language Development in a 

Comprehension-Based ESL Program. 

Jan 92 

22p.; Paper presented at the World Congress of 
Applied Linguistics sponsored by the International 
Association of Applied Linguistics (9th, 
Thessaloniki^ Greece, April 15-21, 1990). 
Reports - Research/Technical (143) — 
Speeches/Conference Papers (150) 



EDRS PRICE 
DESCRIPTORS 



MFOl/PCOl Plus Postage. 

Classroom Techniques; College Students; ^English 
(Second Language); Foreign Countries; Higher 
Education; Instructional Effectiveness; Language 
Proficiency; Language Tests; ^Listening 
Comprehension; ^Reading Cotrprehension; Second 
Lr.nguage Learning; *Second Languages; Self Evaluation 
(Individuals); Teaching Methods; Vocabulary 
Development; Young Adults 



ABSTRACT 

A study investigated the role of comprehension of 
meaningful language input in young adults' second language learning, 
focusing on: (1) what kinds of measurement instruments and procedures 
can be used in tracking student gains in specific aspects of target 
language proficiency; (2) development of a reliable self-report scale 
capturing different vocabulary knowledge levels; and (3) use of 
introspection in exploring links between comprehension of meaning and 
acquisition of vocabulary . Subjects were 37 unive-rsity students in 2 
language classes, an experimental group using a comprehension-based 
approach and authentic texts and a control group using an integrated 
4-skills approach and emphasizing grammar instruction. A series of 
tests were administered: baseline global language proficiency tests, 
two rational deletion cloze tests, a self-report of vocabulary 
knowledge, two measures of grammatical knowledge, and (in the 
comprehension-based class) an introspective task on vocabulary 
acquisition. Results indicate superior gains in the 
comprehension-based class in text comprehension and discourse 
processing, despite smal?er gains in grammatical knowledge, 
suggesting that explicit grammar instruction is not needed to advance 
comprehension skills. However, accuracy in writing and speaking was 
found to be linked to grammar instruction. It is concluded that 
comprehension-based instruction develops receptive proficiency, but 
other gains appear idiosyncratic. (MSE) 



* Reproductions supplied by EDRS are the best that can be made 

* from the original document. 



9 



A Methodology for Studying the Relationship Between 
Comprehension and Second Language Deve'opment in a 
Comprehension-Based ESL Program 

T. Sima Paribakht and Marjorie Bingham Wesche 
University of Ottawa 



Second Language Institute 

University of Ottawa 
600 King Edward Avenue 
Ottawa, Ontario KIN 6N5 
Canada 
(614) 564-2973 



"PERMISSION TO REPRODUCE THIS oh«. otE"c.,,on.i ch .«<i .mp,ov.m.M 

MATERIAL HAS BEEN GRANTED BY educational RESOURCES tNrORMATlON 




TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



2 

BEST COPY AVAiLABLE 



1 

A Methodology for Studying the Relationship Between 
Comprehension and Second Language Development in a 
Comprehension-Based ESL Program* ^ 

T. Sima Paribakht and Marjorie Bingham Wesche 
University of Ottawa 
January 1992 

Introduction 

This paper reports on the first stage of a long-term project to investigate the role of comprehension of 
meaningiiil language input in second langauge development by young adults. The context is a 
comprehension-based program for beginning to intermediate learners of English as a second langauge. 
Our objective is to study the language learning outcomes of a teaching approach which emphasizes global 
comprehension of information-rich written and oral texts. These outcomes include gains in global reading 
and listening comprehension ability, and in vocabulary, grammatical and discourse knowledge. In this 
context we have begun to look at differential vocabulary learning outcomes from written texts which have 
been used for global comprehension in instructed versus uninstructed texts. have begun to try to link 
given learning outcomes to different types of instiiictional tasks used in t».e comprehension-based 
methodology, based on observations of how learners carry out these tasks. In a subsequent phase of the 
r&search we will focus on lexical items in an attf;mpt to establish links between input features, pedagogical 
tasks and the internalization of knowledge about and the ability to use specific forms. Our immediate 
concern, however, has been the development of an appropriate methodology for beginning to investigate 
these relationships. We will discuss our findings to date in the framework of a pilot study carried out 
lA the fall of 1989. 

Theoretical Background 

A widely held view in the current literature on second language acquisition is that one major way in 
which second language learners acquire grammatical and other kinds of language knowledge is through 
exposure to and comprehension of meaningful oral or written input in that language. Considerable 
research has been done on the nature of language input to second language learners (cf. reviews in Hatch 
1983; Chaudron 1988a; Larsen-Freeman and Long 1991; Wesche, forthcoming). Particular attention has 
been given to modifications made by native speakers addressing language learners, which are thought to 
increase the comprehensibility of the input and possibly to facilitate their language acquisition in several 
ways. Some studies have examined the effect of input modifications on the comprehensibility of spoken 
discourse by native speakers (e.g.. Head 1973; Long 1985; Chaudron 1985, 1988). Other research has 
linked such modifications to global proficiency gains (e.g., Edwards et al. 1984; Wesche and Ready 
1985), and in some cases to the acquis! on of specific elements of syntax and vocabulary (e.g., Wagner- 
Gough and Hatch 1975; Lightbown 1983; Hawkins 1987; Braidi 1991). While links between input 
features and acquisition have been den-.onstrated to some extent, there remains a need for research on the 
direct relationship between the act of comprehension and the internalization of linguistic knowledge 
(Faerch and Kasper 1986; Long 1983). Recent work by Ellis (forthcoming) on a methodology for 
investigating the acquisition of syntactic knowledge through meaning-focused comprehension activities, 
using manipulated language, is one of few examples of the latter. Our research is an attempt to link 
comprehension of natural written and spoken language input with aspects of second language development 
in a classroom situation. 
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Comprehension-Based Second Language Instruction 

Theoretical rationale 

The theoretical rationale for a comprehension-based approach at early stages of teaching second languages 
came initially from studies of first language acquisition, in which the development of listening 
comprehension ability precedes and underlies the development of speaking ability. Likewise later, in 
school, fluent reading of age appropriate texts precedes and facilitates the development of composition 
skills. More recently, support for a comprehension-based approach to second language teaching has come 
from empirical research (Postovsky 1974; see reviews in Krashen 1981, 1985). An initial second 
language "incubation" or " vernalization" period, concentrating on the comprehension of meaningful 
language input, is thought to help learners to "formulate a map of meaning and form in their minds and 
to internalize the associations between form and meaning" (Swaffar 1986: 9). If learners are allowed to 
focus on the language as a whole as a vehicle for meaning, it is thought that they will gradually assimilate 
syntactic and vocabulary knowledge in the target language. This receptive knowledge will form the basis 
for the production of utterances in the language. Some researchers go so far as to claim that production 
skills will "emerge" without explicit practice, as students receive and internalize large quantities of 
comprehensible input (Krashen 1985; Swaffar and Stephens 1981). Requiring learners to produce in early 
stages k seen as not only frustrating and anxiety-causing for them, but also as counterproductive to an 
efficient learning process (Postovsky 1974; Terrell 1982). It is thought that forcing production in the 
absence of adequate language knowl^xlge causes students to fall back on first language knowledge, leading 
to the LI and L2 "transfer" phenomena observed in their speech and writing (Newmark 1966), and 
possibly even to the fossilization of inaccurate forms (Krashen 1985). Following this rationale, 
instruction in the comprehension-based approach begins with a period during which the learner is required 
only to comprehend written and oral texts, and not to produce them - although production !j encouraged. 
The incubation period in the comprehension-based approach is claimed to result in more accurate 
production, as learners are given sufficient exposure to native speaker models and enough time to 
internalize accurate oral and written representations of the forms and structures of the target language. 
Empirical evidence has demonstrated that extensive listening before production results in more accurate 
pronunciation (Neufeld and Schneiderman 1980; Postovsky 1981). Less is known about the eff^. 
early emphasis on comprehension in the development of grammatical, vocabulary and discourse 
knowledge. Krashen (1985) suggests that second language grammar acquired through understanu...^ 
of meaning and that initial emphasis on formal aspects of language is counterproductive; form is acquired 
through focus on meaning. There is considerable empirical evidence from Canadian French immersion 
programs and other contexts where the comprehension of meaningfiil input is emphasized almost 
exclusively in the initial stages that such development takes place (Edwards et ai 1984; Genesee 1987; 
Harley 1986, forthcoming; Lightbown 1989; Swain and Lapkin 1982), 

Recent research, including that with French immersion students, suggests, however, that focus 
on form in a meaningftil context may be important in developing grammatical accuracy in some aspects 
of speaking and writing - notably those that lack salience in communicative language use (Allen et al. 
1987; Burger 1989; Harley 1989, 1990; Swain 1985, 1988; Van Patten 1990) and may even enhance 
comprehension (CourchSne and Pugh 1986; Champ agrje, rc;search in progress). Proponents argue that 
learners need to know how formal structures organize and represent meaning, and that learners should 
therefore be made explicitly aware of the links between grammatical structure and meaning in oral and 
written input. 



4 



The ESL programs of the Second Language Institute (SLI) at the University of Ottawa offer the 
possibility of exploring the effect of meaning-focused comprehension-based instruction on second 
language development and, as well, to contrast this with the effect of explicit form-focused instruction 
at similar proficiency levels. The initial research has explored both issues. 

Instructional Context and Procedures 

At the University of Ottawa, a large bilingual (French - English) university in Canada's national capital^ 
undergraduate students are required to reach a certain level of listening and reading proficiency in their 
second language before graduation. (Their chosen language of instruction is considered to be their first 
language.) In response to the institutional needs of students^ the SLI has developed a comprehension- 
based program comprising 4 one-semester courses, drawing on the theoretical and empirical bases 
outlined above. The courses, given in both English and French for begimiing and intermediate students, 
emphasize comprehension of meaningful oral and written texts throughout. The program consists of a 
progression of reading and listening skills to be mastered (e.g., identifying the main idea, scanning, 
inferencing) and a bank of authentic, thematically organized teaching materials for both reading and 
listening. The themes are carefully selected for their interest and appropriateness for university students, 
their current lopicality, the availability of material and their potential for exploitation in the classroom. 
There is no systematic teaching of grammar in these courses, although grammatical questions and 
problems which arise are dealt with by teachers in context. Oral and written production in the second 
language is not required, but is encouraged as the student progresses so that by the end of the second or 
early in their liiiird semester students may frequently speak or write in class. They are not evaluated on 
this, however. The parallel existence of four-skill courses with explicit grammar instruction for other 
clienteles at the same proficiency levels at the University offers the possibility of comparative study of 
language development in the two instructional contexts. 

A pilot study was carried out with two intermediate level ESL courses using different teaching 
approaches. These one-semester classes included 54 contact hours (four hours per week). 

Research Issues 

Methodological Questions 

The methodological issues dealt with in the pilot study were the following: 

What kinds of measurement instruments and procedures can be used to track student gains in 
specific aspects of target langauge proficiency (e.g,, vocabulary, syntax, discourse markers)? 

Can a reliable self-report scale be developed which captures different levels of knowledge of 
vocabulary items? 

Can introspection be used as a technique for exploring links between comprehension of meaning 
and acquisition of vocabulary, and if so, how? 
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Theoretical Questions 

Working hypotheses were formulated for this research in the form of general and specific questions': 

1) In which aspects of English language proficiency are gains made in Comprehension-based 
classes (e.g., in global receptive proficiency, vocabulary, grammar)? 

2) Which type of vocabulary items are most easily acquired, content words (nouns, verbs, 
adjectives) or discourse connectives? 

3) Are content vocabulary items which are related to instructed themes more readily acquired than 
those taken from uninstructed themes? 

4) Can different stages of acquisition of vocabulary items be identified? 

5) Is there an identifiable progression in the development of sentence-level grammatical knowledge? 
That is, does the ability to recognize correct or incorrect sentences precede the ability to correct 
the incorrect sentences? 

6) Is grammatical knowledge acquired in comprehension-based courses without explicit grammatical 
instruction? 

7) How do the results form a Comprehension-based course differ from those of a Four-skill course 
at the same level? 

Methodology 

Subjects 

Subjects were 37 young adult university students (19 in the Comprehension-based class in 18 in the 
Four-skill comparison class) from a mixture of first langauge backgrounds.* Both groups included a 
range of intermediate proficiency levels, as assessed by a placement test at the beginning of the session. 

Classroom Treatment 

The Comprehension-based class focused on the development of listening and reading skills using 
authentic texts on selected themes. There was no systematic grammatical instruction. Out of 4 hours of 
weekly class time, approximately 1 '/i hours were spent on instruction and practice of reading skills 
(e.g., scanning, skimming, guessing meaning in context, recognition of paragraph patterns) and the 
reading of authentic texts and answering comprehension questions on their content. Another 1 '/i hour 
period of class time was reserved for comprehension practice in the language laboratory, during which 
students would listen to an authentic audio or video recording on the same theme as the current reading 
text and would then answer a series of questions focusing on different aspects of comprehension (global, 
specific information, inferencing). One hour per week was spent on newspaper or magazine reading and 
further development of reading skills with these texts. 

The Four-skill class was taught using an integrated four-skill approach with 1 V2 hours a week 
devoted to grammar instruction, including explicit exercises on prepositions and vocabulary. Theme- 
based listening and reading materials, including 2 themes which were also taught in the exj'erimental class 
(i.e., Media and Environment) were used for up to V2 hour a week. A further '/^ hour a week was 
devoted to writing activities and at least that much time to organized speaking activities. 
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Measures 



The measurement instruments for this study were all based on written texts or sentences. The following 
measurement instruments or tasks were selected or developed for the study, and were administered at the 
beginning and end of the course unless otherwise specified. 

Global Receptive Pronciency Tests 

To establish base-line data, a Placement test of listening and reading skills was administered to both 
groups at the beginning of the course. Unfortunately, due to class-time constraints and program 
requirements, it was not possible to re-administer the Placement test to the two groups as a measure of 
gains in global receptive proficiency. A similar Exit test was, however, given to the Comprehension- 
based class at the end. The Placement and Exit tests include reading and listening sections with questions 
and activities based on authentic texts (e.g., dialogues, news stories, instructions, explanations, short 
reports, interviews or lectures). There are a variety of response formats including chart filling, short 
written answers, true or false and multiple choice. Questions test global comprehension of the text, as 
well as the ability to, for example, identify main and supporting ideas and to make inferences. 

Vocabulary Measures 

Three themes were selected as a basis for instrument development, according to the criteria discussed 
above. One of the themes. Media, was used for instructional units in both the Comprehension-based 
and the Four-skill classes. A second theme, Family Custody, was used for instruction and introspective 
data gathering in the C<»mprehension-based class, and a third theme. Fitness, was retained for 
comparison testing (i.e., to measure student progress with materials not related to instruction). 

A number of theme-related content words (nouns, verbs, adjectives) and cohesive markers 
(e.g., "in fact", "because", "however"), were selected from each written text. These items were then 
screened for French cognates with the help of a bilingual native speaker of French, since it is known that 
cognates present siveciai learning characteristics and are generally easy for students in our bilingual 
context (Duquette 1991). Based on the selected items, two sets of instruments were then developed to 
measure vocabulary acquisition, as follows: 

Cloze Tests 

Two rational de!aion Cloze tests with approximately 35 blanks each were developed. These were created 
from theme-related written texts selected for the study (i.e., Media and Fitness), and tested the selected 
content words and cohesive markers. Students were asked to fill in the blanks from a master list whicf 
included ail the missing words plus five d'stractors. The original (unmutilated) media text was presented 
in cia.ss as part of the theme unit on media, to develop comprehension skills and to provide general 
informational background. 

Vocabulary Knowledge Scale 

A self-report Vocabulary Knowledfte Scale (VKS) was developed for the study (see Figure 1). This 5- 
level descriptive vocabulary knowledge scale represented an attempt to capture different levels of self- 
perceived knowledge of specific words. These levels range from total unfamiliarity through recognition 
of the written word and some idea of its meaning, to the ability to use the word in a sentence. 
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Figure 1 Vocabulury Knowledge Scale Categories 
I. I have never seen this word. 

If. I have seen this word before, but I don't know what it means. 

III. I have seen this word before, and I think it means . (synonym or 

translation) 

IV. I know this word. It means . (synonym or translation) 

V. I can use this word in a sentence: . 

Students were administered each of the three theme-related word-lists and the corresponding VKS 
on a separate occasion. (Listed words were presented in scrambled order to remove contextual 
information.) Students were then asked to indicate on the VKS their degree of knowledge of each target 
word. The VKS lists were administered prior to students' taking the corresponding cloze test. 

VKS Scoring 

The student self-report of vocabulary knowledge in categories IIMV on the VKS was used as evidence 
of demonstrated rather than perceived knowledge. The following scoring adjustments were thus made 
to the self-reported knowled;.^ categories, resulting in a 6-level scale. 

Category III: If the synonym or translation given was clearly wrong, the response was scored 
as category II; if correct, it was scored as category IV. Thus category HI was important for eliciting 
perceptions and evidence of vocabulary knowledge for two different scoring categories. If the student 
indicated category V and produced a sentence from which it was impossible to tell if he or she knew the 
exact me;^ning or not, it was scored as category III. The following examples were taken from student 
responses for the word "wildlife", a student wrote "She shall love wildlife."; for the word 
"masterpiece", another wrote "The fire destroyed the masterpiece of the city." Both were scored as 
category III. 

Category V: It was necessary to split category V into two levels, to reflect both semantic 
knowledge and grammatical exactness of the use of the target word in a sentence context, if the word 
was mis-used according to both criteria, it was scored as category II. A category V score reflected 
knowledge of the referential meaning of the target word, but its use in a context requiring a different 
grammatical category (e.g., a target noun used as a verb form). Examples for the target word 
"chemicals" were: "Hydro Quebec produce some dangerous chemicals liquid with electricity," and "I 
work for a chemical products company." An example for "to retire" was "This famous player announce 
his retire." Other grammatical problems in formation of the sentence were ignored. Scoring category 
VI reflected both semantically and grammatically correct use of the target word in a sentence. 

Grammar Tests 

Two measures of grammatical knowledge were used in the study. The first, a test of grammatical 
knowledge developed by Bialystok and her colleagues (1981), consisted of five correct and twenty 
incorrect sentences (the latter with errors of tense, number, word order, etc.). Students were given 
scores for two levels of grammatical knowledge: I) grammaticality judgements (recognition of correct 
and incorrect sentences) and 2) for their ability to correct the incorrect sentences.^ 
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The second test measured knowledge of common Prepositions (i.e., at, on, to, with, in, of, for). 
In this test students were required to fill in missing prepositions in twenty sentences.^ Scores were based 
on the number of correct prepositions provided. 

Introspection Task 

Fuaher data was gathered on vocabulary acquisition in the Comprehension-based class. A retyped 
newspaper text on the theme of Family Custody in a divorce case was given in class as a pre-test for 
vocabulary icnowledge. Students were asked to read tlie text rapidly and to underline all words whose 
meanings they didn't know. The students were then given another copy of the text, this time divided into 
segments of several sentences each. They were asked to read the text again and to answer six 
comprehension and four inferencing questions. While doing this, they were asked to write (in the spaces 
between segments) how they dealt with each unfamiliar word they had identified (e.g., ignore it^ try to 
infer the meaning, look it up). A few days later, each student was given the list of words he or she had 
identified as unknown, together with the Vocabulary Knowledge Scale, and asked to indicate his or her 
level of knowledge of each word. The data collected included scores for the reading comprehension 
exercise, the questionnaires and the comments. The introspective data was gathered only in the 
Comprehension-ba.sed class, during the 8th week of a 13 week semester. 

Analyses 

The statistical analyses included calculation of: 
descriptive statistics on all variables; 

analyses of covai iance for all pre-post variables to determine whether there were gains f either 
group or any between-group differences in patterns of gains. (In the first case, significant group 
gains are reported from T-tests as well.) 

Frequency data wei'e calculated on the VKS for all words, to determine the levels of vocabulary 
knowledge demonstrated by each individual for each item at the beginning and end of the course. They 
were also calculated for discourse connectives versus content words separately. Finally, to allow 
multivariate analysis of vocabulary gains, the pre-post status of each word on the VKS was scored 
dichotomously. To do this. Vocabulary Knowledge Scale categories I, II and III (levels of relative 
familiarity with words for which the correct meaning is not known) were grouped into a "^not known"* 
category, and categories IV, V and VI (levels of knowledge of word meaning and appropriate use), were 
grouped into a known" category. Scores were calculated for each student on the total number of 
''known" woids at the beginning and end of the course, as well as for content words and discourse 
connectives. Pre-post gain and Ancova analyses were done on these ("known"/"not known") scores as 
well. 

Findings 

Methodology 

With respect to methodology, the measures we used in our initial study proved largely appropriate for 
the intermediate proficiency level of our subjects and for our research purposes. All measures are 
sensitive enough to reflect specific gains on different aspects of language knowledge and use, but certain 
retlnements are required. These are discussed below with respect to each instrument. 
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1 . Global Receptive Proficiency Tests 

The Placement and Exit tests used in the study are specifically designed for the comprehension-based 
courses and thus are of an appropriate difficulty level for these students. The students in the 
comprehension-based class were also exposed to classroom tasks similar in format to those appearing on 
these tests. Whil?: the comparison class had some exposure to such activities, there may be some 
advantage on thes^. tests for the comprehension-based students. In a future comparative study, a more 
rnethod-neutral test should be used for both groups at the beginning and end of instruction, which presents 
a balance of tasks better reflecting the classroom activities of both groups. 

The major problem with testing global proficiency was logistical - due to miscalculation of the 
time demands on participants in our study - and resulted in an unforeseen critical gap in the comparison 
data. Since it turned out to be impossible to administer the Exit test to this group during class time, the 
only possibility would have been to pay students to return after their final examination. This was not 
feasible in the exploratory study, but should be built into a future comparison. 

2. Vocabulary Knowledge Scale 

Students appeared to answer the VKS with certainty, and the patterns of change in vocabulary knowledge 
of the target words during the course indicate that it captures something of a progression in development 
of vocabulary knowledge. There appeared to be some variation in student interpretation of "I think" and 
"I know" (categories III and IV). Also, words were sometimes used in general ways in category V 
sentences which made it difficult to judge whether or not students knew a specific meaning of the word. 
To correct these problems we have clarified the instructions for the VKS by emphasizing the key 
instruction words, and now require that students writing a sentence for category V also provide a meaning 
in category IV. These changes should improve scale reliability. We plan to establish the reliability of 
the scale with a large, independent sample of ESL students. We also will seek to establish the relative 
distance between the VKS categories, using dual scaling. 

The validity of the VKS also needs to be fully examined; however, this is difficult given the 
dearth of vocabulary instruments which aim to measure growth in knowledge of individual words (Curtis 
1987). In any case, we intend to carry out a concurrent validity study with a general proficiency measure 
and if possible with a measure of vocabulary breadth with the independent sample. 

It was unfortunate that we had to merge the VKS categories into two major categories ("known" 
and "unknown" words) for quantitative analysis, as gains between adjacent categories could not be 
demonstrated statistically with this sample size (suggested by the descriptive statistics. Table 2). This can 
best be remedied through a much larger sample. 

3. Cloze Tests 

We used texts appropriate for intermediate level students to construct the Cloze Tests, which consequently 
tended to be too long and too difficult for this level due to the reduced redundancy. The use of a word 
list probably made the task harder and more time consuming than an open cloze, although it facilitated 
reliable scoring. Since the time requirements of these tests made it almost impossible to administer all 
of them during class hours, students in the comparison group were assigned some of them at home, which 
resulted in cheating and invalid data. 
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In order to avoid this problem and to ensure that all students do all the assignments under similar 
conditions, we are now developing summary cloze tests with fewer words and accompanying word lists. 
(/V multiple choice format, with distractors based on errors made by similar groups of students, might 
provide an alternative solution. It would be an easier task, but the disadvantage is that recognition rather 
than recall would be emphasized.) 

4. Grammar Tests 

The preposition test appeared to work well. We plan to establish its statistical reliability. 

The grammar test included S correct and 20 incorrect sentences. Based on the pilot study we are 
developing a new instrument which will include the same number of correct and incorrect items. We will 
specify grammatical points to be tested, and prepare parallel correct and incorrect items for each. For 
example, an item such as "Water is extreme important for life." would be balanced with an item such 
as "The cubs are generally born m the early spring." 

After establishing the reliability of the test, we would then be able to adequately test the implied 
hierarchy of grammatical knowledge, as follows: 

Level 1 : correct items judged as correct. 
Level 2: incorrect items judged as incorrect. 
Level 3: incorrect items corrected. 

5. Introspection Task 

The introspection task yielded interesting information about the influence of different comprehension tasks 
on student learning behaviour, i.e., what ^^tudents pay attention to. We intend to work furuier on this 
procedure as well as to explore other introspection techniques as a means of seeking more detailed 
indications of mental processes involved in performing comprehension tasks. It appears to us that some 
training of students to analyze and discuss their learning processes would be help<^al The introspection 
exercise in the pilot study involved writing comments. We wish to supplement ihi» with recorded oral 
protocols in the future. We would also allow them to do the exercises in their mother tongue. 

The methodology in general is promising for written language inL-ut v\(i outcomes, and parts of 
it could probably be adapted for work on listening and speaking. 

Theoretical Outcomes 

Tables I and 2 present descriptive statistics on each variable for all subjects having valid pre-test and 
post-test scores, results of T-tests showing the statistical significance of pre-post gains, Ancova analyses 
of differences in gains between the comprehension-based and four-skill classes, and bar graphs illustrating 
pre-post scores on the VKS. These are discussed below. 

Gains were made in receptive English proficiency by the Comprehension-based classes and 
comparison group in global receptive proficiency, vocabulary and grammar following somewhat different 
patterns. 
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Global Receptive Proficiency 

Gains in overall receptive proficiency are apparent for both groups from indirect evidence, even though 
it was not possible to use the same pre-post measure. This evidence includes teacher observations and 
class tests, and the large difference in pre- and post-course percentage scores on measures of similar 
difficulty for the Comprehensicn-based class (Placement and Exit tests). 

Vocabulary 

Both classes showed gains on at least some vocabulary measures, revealing somewhat different patterns 
on the cloze tests and VKS favouring the comprehension-based class. There were also differences in 
gains between content words and discourse connectives, the former in general being far eaaier, and, as 
one might expect, better learned when part of thematic class instruction. On the VKS word-lists for the 
instructed theme, a comparison of pre- and post-course results showed gains for both classes (total score 
and content word score). However, only the Comprehension-based class showed significant gains on 
discourse connectives. On the uninstructed theme, for which data were available only for the 
Comprehension-based class, VKS gains were significant for the total score and content words but not 
for discourse connectives. These results are illustrated graphically on Table 2 and reflect shifts from self- 
report categories I-IH (word not known) to categories IV-VI (word known) on the subscores for this 
scale. While it was impossible to statistically verify progress of students on individual words from one 
scale category to another, the bar graphs indicate that such progress was made. 

The Comprehension-based class made highly significant gains on the instructed cloze test 
(Media) on measures of total "known** words, content words and discourse connectives, while on the 
uninstructed cloze test (Fitness) gains were only significant for content words. The Four-skill class, on 
the other hand, made significant gains only on the content words on the instructed cloze test. 
(Unfortunately the dat? for this class on the uninstructed theme were not valid due to the procedural 
problems mentioned earlier.) While these results suggest different patterns of gains for the two groups 
and stronger gains by the Comprehension-based class, there were no significant inter-group differences 
related to the Cloze and VKS vocabulary measures (Table 1). 

Grammar 

On the grammar tests, neither class showed significant gains on the ability to recognize correct or 
incorrect sentences. The Four-skill class, however, had significantly higher scores at the end of the 
course or !:e ability to correct incorrect sentences, and significantly more than the Comprehension- 
based class. 

Both groups made significant gains on the preposition test. Again the gains of the Four-skill 
class were significantly greater than those of the Comprehension-based class. Thus while there is some 
evidence that grammatical knowledge is acquired in comprehension-based courses without systematic 
grammatical instruction, such instruction appears to lead to greater gains on these measures. 
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Fitness data 

not available 
for this class 
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Introspection Task 

It was clear from student responses to the introspection task that their most common response to unknown 
words was to ignore them, unless understanding was specifically required to answer the comprehension 
questions after the reading task. Guessing the required meanings (sometimes with the help of the context 
in the comprehension question as well as in the text) was the major strategy used with needed words. 
Some students reported looking in a dictionary - a strategy used more by certain individuals than others. 
All in all these findings suggested tha^ students followed the "law of minimum effort" and that the nature 
of the comprehension questions determined to a large extent the extra mental effort they would make to 
find meanings of unknown vocabulary *tems. 

Discussion 

VAeoretical Issues 

The VKS results indicated that the Comprehension-based class made significant gains in knowledge of 
content v;ords and discourse connectives relating to the instructed themes, whereas the Four-skill class 
demonstrated gains only on content words, and not on discourse connectives. (Both groups also made 
significant gains on the "total words" measure, mainly reflecting gains on content word knowledge.) 
These differential outcomes may indicate that the greater focus in the Comprehension-based class on 
comprehension of the meaning of discourse chunks, which may reinforce (by making more salient) the 
meaning and functions of words related to text organization. 

The VKS results ftirther indicate that significant gains by the Comprehension-based class in 
knowledge of discourse connectives were only in relation to the instructed theme and not the uninstructed 
one; i.e., students were better able to provide the discourse connectives in the instructed text. This rv^y 
simply be due to their better comprehension of the instructed text that facilitated their performing the task. 
Another possibility is that while discourse connectives are not theme specific, certain discourse 
connectives may be more characteristic of given text types* (e.g., a scientific text may exhibit more cause 
and effect related discourse devices), and these text types may be associated more with certain themes. 
Replication of this finding and appropriate frequency counts for discourse connectives of interest in 
relation to all texts presented would be required for fdrther interpretation. 

The Comprehension-based class likewise made significant gains on the cloze test based on the 
instructed theme for total words, content words and discourse connectives, while on the uninstructed 
theme gains were only significant for content words. This nr.ay again suggest one way in which 
instruction makes a difference. Instructional focus on the content, organization and theme-related 
vocabulary of a given text, leading to better comprehension, appears to aid students in making a more 
successful choice of words for cloze blanks on the text at a later time when they no longer remember 
specific facts about the passage. The Four-skill class did not show significant gains on the discourse 
connectives, even though both classes spent approximately the same amount of time on the instructed 
theme. This probably reflected less focus and practice on text comprehension strategies. In spite of the 
different gain patterns for the two groups on vocabulary measures, inter-group differences were not 
significant. This may have been due to the extreme heteiogeneity of both classes in proficiency level - 
which is reflected in all measures used in the study. Again, replication with more homogeneous groups 
is needed. 
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Evidence was found of gains in grammatical knowledge (prepositions for both groups and 
grammatical correction skills for the Four-skill class). Comprehension-based class gains on knowledge 
of prepositions and discourse connectives provide evidence that some (lexically based) grammatical 
knowledge can be acquired incidentally through an exclusive focus on comprehending meaningful 
language without grammatical instruction. The considerable gains of the Comprehension-based class 
in knowledge of prepositions may be due to the fact that prepositions are structure/fiinction words. It 
may be that emphasis on and consequent gains in text comprehension in the Comprehension-based class 
result in better understanding of the meaning and Unction of these words. However, superior gains of 
the Four-skill class suggest that explicU practice in the recognition and use of prepositions leads to a 
higher level of mastery. 

There was no evidence of gains in grammatical ity judgement (the ability to judge correct versus 
incorrect sentences) for either group. It is possible that both groups had indeed improved in this respect 
(i.e., implicit grammatical knowledge) but 1) due to the relatively high pre*course scores and the small 
number of sentences, change was not demonstrated statistically. On the other hand, the Four-skill class 
gained significantly on the ability to correct incorrect sentences, and significantly more than the 
Comprehension-based class. 

The fact that gains on several grammatical measures (prepositions and error correction) were 
significantly greater for the Four-skill class suggests that explicit grammar teaching enhances students' 
ability to provide grammatically correct forms on tasks which draw their attention to these forms. (We 
did not in this study look at the quality of spontaneous grammar use, but intend to do so at a later stage 
of the project.) 

Preliminary results from the introspection study demonstrated clearly that students tend to do the 
minimum required. Thus in comprehension exercises they seek or infer meaning of new words in reading 
texts only if this is necessary to answer the questions at the end. Our pedagogical conclusion is that the 
nature of the tasks students are required to do strongly influences what they will attend to, and should 
therefore be a very important consideration for the classroom teachers. 

There is also some evidence of gains in productive vocabulary skills by both classes (i.e., the 
number of students scoring in *'known" categories IV, V and VI on the VKS). We will elicit and 
examine gains in productive skills more systematically in our fiiture research. 

Conclusion 

Student gains on different measures based on the instructed (vs. the uninstructed) theme, as well as 
superior gains in grammatical knowledge for the comparison group clearly indicate that instruction makes 
a difference in the level of gains on different aspects of language knowledge. This conclusion is further 
supported by the instrospection data indicating that the nature of instructional tasks affect what learners 
attend to and consequently may eventually learn. 

The results of the study suggest superior gains by the Comprehension-based class in text 
comprehension and discourse processing, despite their lower level of gains in error correction 
(grammatical knowledge). It may be reasonable to conclude, therefore, that explicit grammar instruction 
is not needed for the improvement of text comprehension skills. At the same time, when learners are 
focused on meaning, they may not attend grammatical forms, particularly those not required for global 
understanding, and may not, therefore, substantially improve their grammatical knowledge. The superior 
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results of the Four-^kill class on grammatical measures suggest that learners may need to receive some 
instruction focused on grammatical forms in order to develop sensitivity to them and as a basis for greater 
accuracy in speaking and writing. 

It appears that a comprehension-based program can achieve its instructional objectives in 
developing learners* receptive proficiency. Gains in other aspects of language are incidental and may 
exhibit idiosyncratic patterns among individual learneis (e.g., some students may analyze formal aspects 
of language even though these are not a focus of instruction). A comprehension-based ESL program 
appears to be a viable instructional alternative in academic contexts where priority is accorded to the 
development of receptive skills for high level text and lecture comprehension, and may also be a solid 
preparation for moving into more form-focused instruction and productive use of the target language. 

We intend to follow-up this study with a series of within-class Studies, limited to comprehension- 
based courses and focusing on vocabulary growth in this context. This research will incorporate the 
methodological improvements indicated above, including elicitation techniques for gathering information 
on students' ability to use the target language in freer written production tasks. 

We intend to use both pre-post measurement and introspective techniques in the initial within-class 
study, then to replicate that study in a similar class, but alternating the instructed vs. uninstructed themes. 
In subsequent studies we would vary proficiency levels of subjects and, eventually, teaching methodology, 
thus building up a composite picture of classroom processes in the learning of vocabulary and other 
aspects of an L2. 

Notes 

1. Wc arc very grateful lo ihc following [>crsons for ihcir help with ihc project; Manko Rangongo. who taught the 
(our-skill class; Lisc Duquette, who helped in the screening of French cognates for the vocabulary measures; Marie>Jos<5c 
Vignola, Justine Foxall and Karina Fleck, who were our research assistants at various stages and Dorecn Ready, our 
statistician. 

2. An earlier report on this study was presented at the 9th World Congress of Applied Linguistics in Thessaloniki, 
Greece, in April 1990. 

3. Most of these hypotheses are dealt with in this study. Several remain to be explored in subsequent research. 

4. The linguistic composition of the 2 classes was as follows: Comprehension-based (N = 18): 10 francophones, 5 
Arabic speakers, and 1 speaker each of Chinese, Spanish and Comorean; Four-skill (N = 19): 7 francophones, 4 Chinese 
speakers, 3 Polish speakers, 2 Arabic speakers, and one speaker of each of Japanese, Vietnanfiese and Mina. 

5. These arc examples of the type of sentences which api^cared on the grammatieality knowledge test: 

Correct: "Jade is a stone of high value." 
Incorrect: "She speaks French very good." 

6. These are examples of items on the preposition test: 

"She has applied a new job," 

"Exposure violence on TV is harmful to children." 
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